{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3c50a36c",
   "metadata": {},
   "source": [
    "# 交叉验证\n",
    "在交叉验证中，我们使用不同的数据子集来执行建模过程，以获得模型质量的多个度量，例如，我们可以将数据分成5个部分，每个部分占整个数据集的20%。.在这种情况下，我们将数据分成5个“folds”，然后，我们对每个fold进行一次实验：\n",
    "###### 在实验1中，我们使用第一个fold作为验证（保留）数据集，其他fold为训练数据。这给我们一个基于20%保留数据集的模型质量度量。\n",
    "###### 在实验2中，我们保留第二个fold的数据（其他fold为训练数据）。这将得到第二个模型质量评估结果。\n",
    "###### 以此类推，使用每个fold作为验证数据集，我们最终得到的模型质量，是基于所有的行数据集(即使我们不同时使用所有行)。\n",
    "### 什么时候使用交叉验证？\n",
    "###### 对于小数据集，额外的计算并不是什么大事，你应该进行交叉验证。\n",
    "###### 对于较大的数据集，一个验证数据集就够了。你的代码将会执行得更快，有足够的数据就没必要复用数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a8d702b6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rooms</th>\n",
       "      <th>Distance</th>\n",
       "      <th>Postcode</th>\n",
       "      <th>Bedroom2</th>\n",
       "      <th>Bathroom</th>\n",
       "      <th>Car</th>\n",
       "      <th>Landsize</th>\n",
       "      <th>BuildingArea</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>Lattitude</th>\n",
       "      <th>Longtitude</th>\n",
       "      <th>Propertycount</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13333</th>\n",
       "      <td>3</td>\n",
       "      <td>6.3</td>\n",
       "      <td>3013.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>351.0</td>\n",
       "      <td>125.0</td>\n",
       "      <td>1910.0</td>\n",
       "      <td>-37.81111</td>\n",
       "      <td>144.88759</td>\n",
       "      <td>6543.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13334</th>\n",
       "      <td>3</td>\n",
       "      <td>10.4</td>\n",
       "      <td>3042.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>697.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-37.73254</td>\n",
       "      <td>144.88321</td>\n",
       "      <td>3464.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13335</th>\n",
       "      <td>3</td>\n",
       "      <td>10.4</td>\n",
       "      <td>3042.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>626.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-37.71919</td>\n",
       "      <td>144.87842</td>\n",
       "      <td>3464.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13336</th>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3206.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>135.0</td>\n",
       "      <td>1880.0</td>\n",
       "      <td>-37.84398</td>\n",
       "      <td>144.95044</td>\n",
       "      <td>3280.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13337</th>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3206.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>241.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-37.84479</td>\n",
       "      <td>144.94429</td>\n",
       "      <td>3280.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13575</th>\n",
       "      <td>4</td>\n",
       "      <td>16.7</td>\n",
       "      <td>3150.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>652.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1981.0</td>\n",
       "      <td>-37.90562</td>\n",
       "      <td>145.16761</td>\n",
       "      <td>7392.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13576</th>\n",
       "      <td>3</td>\n",
       "      <td>6.8</td>\n",
       "      <td>3016.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>333.0</td>\n",
       "      <td>133.0</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>-37.85927</td>\n",
       "      <td>144.87904</td>\n",
       "      <td>6380.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13577</th>\n",
       "      <td>3</td>\n",
       "      <td>6.8</td>\n",
       "      <td>3016.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>436.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>-37.85274</td>\n",
       "      <td>144.88738</td>\n",
       "      <td>6380.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13578</th>\n",
       "      <td>4</td>\n",
       "      <td>6.8</td>\n",
       "      <td>3016.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>866.0</td>\n",
       "      <td>157.0</td>\n",
       "      <td>1920.0</td>\n",
       "      <td>-37.85908</td>\n",
       "      <td>144.89299</td>\n",
       "      <td>6380.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13579</th>\n",
       "      <td>4</td>\n",
       "      <td>6.3</td>\n",
       "      <td>3013.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>362.0</td>\n",
       "      <td>112.0</td>\n",
       "      <td>1920.0</td>\n",
       "      <td>-37.81188</td>\n",
       "      <td>144.88449</td>\n",
       "      <td>6543.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>247 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Rooms  Distance  Postcode  Bedroom2  Bathroom  Car  Landsize  \\\n",
       "13333      3       6.3    3013.0       3.0       2.0  2.0     351.0   \n",
       "13334      3      10.4    3042.0       3.0       1.0  1.0     697.0   \n",
       "13335      3      10.4    3042.0       3.0       1.0  3.0     626.0   \n",
       "13336      3       3.0    3206.0       3.0       2.0  1.0     162.0   \n",
       "13337      3       3.0    3206.0       3.0       1.0  1.0     241.0   \n",
       "...      ...       ...       ...       ...       ...  ...       ...   \n",
       "13575      4      16.7    3150.0       4.0       2.0  2.0     652.0   \n",
       "13576      3       6.8    3016.0       3.0       2.0  2.0     333.0   \n",
       "13577      3       6.8    3016.0       3.0       2.0  4.0     436.0   \n",
       "13578      4       6.8    3016.0       4.0       1.0  5.0     866.0   \n",
       "13579      4       6.3    3013.0       4.0       1.0  1.0     362.0   \n",
       "\n",
       "       BuildingArea  YearBuilt  Lattitude  Longtitude  Propertycount  \n",
       "13333         125.0     1910.0  -37.81111   144.88759         6543.0  \n",
       "13334           NaN        NaN  -37.73254   144.88321         3464.0  \n",
       "13335           NaN        NaN  -37.71919   144.87842         3464.0  \n",
       "13336         135.0     1880.0  -37.84398   144.95044         3280.0  \n",
       "13337           NaN        NaN  -37.84479   144.94429         3280.0  \n",
       "...             ...        ...        ...         ...            ...  \n",
       "13575           NaN     1981.0  -37.90562   145.16761         7392.0  \n",
       "13576         133.0     1995.0  -37.85927   144.87904         6380.0  \n",
       "13577           NaN     1997.0  -37.85274   144.88738         6380.0  \n",
       "13578         157.0     1920.0  -37.85908   144.89299         6380.0  \n",
       "13579         112.0     1920.0  -37.81188   144.88449         6543.0  \n",
       "\n",
       "[247 rows x 12 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 加载数据 X_train、X_valid、y_train和y_valid\n",
    "import pandas as pd \n",
    "from sklearn.model_selection import train_test_split\n",
    "melb_data = pd.read_csv('../melb_data.csv').loc[13333:,:]\n",
    "y = melb_data.loc[:,'Price']\n",
    "X_col = ['Type','Method','Regionname','Rooms','Distance','Postcode',\n",
    "       'Bedroom2','Bathroom','Car','Landsize','BuildingArea','YearBuilt',\n",
    "       'Lattitude','Longtitude','Propertycount']\n",
    "X_col = [col for col in X_col if melb_data[col].dtype in ['int64', 'float64']]\n",
    "X = melb_data.loc[:,X_col]\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5022f467",
   "metadata": {},
   "source": [
    "我们定义一个pipeline，使用填充器来填充缺失数据，以及随机森林模型来做预测。\n",
    "如果不使用 pipeline，来交叉验证是非常困难的！使用pipeline将会使代码非常简单。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "42a9db3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "my_pipeline = Pipeline(steps=[('preprocessor', SimpleImputer()),\n",
    "                              ('model', RandomForestRegressor(n_estimators=50,\n",
    "                                                              random_state=0))\n",
    "                             ])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba992915",
   "metadata": {},
   "source": [
    "使用cross_val_score()函数来获取交叉验证评分，通过cv参数来设置fold数量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "cec19c3d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE scores:\n",
      " [387957.3696     249804.4        260100.68408163 322910.93142857\n",
      " 229306.62612245]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# 乘以-1，因为sklearn计算得到的是负MAE值（neg_mean_absolute_error）\n",
    "scores = -1 * cross_val_score(my_pipeline, X, y,\n",
    "                              cv=5,\n",
    "                              scoring='neg_mean_absolute_error')\n",
    "\n",
    "print(\"MAE scores:\\n\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "447745b4",
   "metadata": {},
   "source": [
    "scoring参数指定了模型质量的度量方法，我们选择负的平均绝对误差值，指定负MAE有点奇怪。Scikit-learn有一个约定，大的数字意味着更好的效果。在这里使用负号可以使它们与约定保持一致。\n",
    "\n",
    "###### 我们通常需要模型质量的单一度量来比较不同的模型，所以取全部实验的平均值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "e45a808d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average MAE score (across experiments):\n",
      "290016.0022465306\n"
     ]
    }
   ],
   "source": [
    "print(\"Average MAE score (across experiments):\")\n",
    "print(scores.mean())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
